Run ate-api-server in HA mode#538
Conversation
803cf91 to
bf29da6
Compare
We have a few places where we need to do port-forwarding to Pods behind a service. Consolidate this logic under `internal/portforward`.
This demo is broken now that we require authn to ate-apiserver. The fix is not trivial as it requires the actor to be able to authenticate, plus it doesn't really adds much.
Holding state in memory means that we cannot scale to more than one replica. Change the mock implementation to be stateless. It doesn't really need to track state anyways.
e93b02c to
321dcdb
Compare
| // periodically re-resolve DNS and pick up replicas added since they | ||
| // last connected - without this, an existing connection never | ||
| // notices new replicas on its own (see https://github.com/grpc/grpc/issues/12295). | ||
| grpc.KeepaliveParams(keepalive.ServerParameters{ |
There was a problem hiding this comment.
This is a bit concerning to me, effectively we are trading off session stability(1m keepalive) in order to obtain the endpoints. I see in the referenced issue that people get periodic unavailable errors with this approach. From the aforementioned issue:
The Go client has this resolver thing. You can import the package and Register a resolver builder or whatever. The client automatically picks it up, so I guess it's a global.
My coworker hacked something together to watch endpoints in the Kubernetes API and we have perfect load balancing when pods go in an out of service.
I think this is fine for now as we iterate, but seems like there's a way to pass endpoints into the resolver. We can probably make this a follow up bug, I don't mind hacking on this if wanted :)
There was a problem hiding this comment.
I think we will eventually end up implementing our own resolver that watches endpoints. Let me add a TODO comment.
A ClusterIP Service load-balances per TCP connection, not per RPC, but gRPC multiplexes many RPCs over one long-lived HTTP/2 connection, so a client that dials once (e.g. ate-controller, atenet) gets pinned to whichever single pod that connection was routed to, for its entire lifetime. Make the "api" Service headless so DNS returns one A record per ready pod instead of a VIP, and switch clients to a "dns:///" dial target with the round_robin load balancing policy so RPCs actually spread across every replica.
This is the minimum HA setup we can run. Configure rolling update policy to recreate replicas one by one, make before break. Also configure PDBs so that we never lose more than one replica during pod eviction events.
321dcdb to
7d34fb1
Compare
2 participants
This PR updates
ate-apiserverdeployment to run with 2 replicas.Some changes were needed to enable this, and are separate in a series of independent commits.
kubectl-atebinary was not checking for Pod readiness. With multiple replicas, there is a risk of picking a not ready Pod and failing the request.agent-secretdemo: This demo is broken now thatate-apiserverrequires authn. I don't see an easy way to fix it, and the demo was not really adding much value.ate-apiserverdeployment toreplicas: 2: Additionally, configure the deployment rolling update so that we update one instance at a time (make before break) and PDB so that we never lose more than one replica during eviction events.Part of #181